Sector- Focused Advancement Strategy 

Two-Year Impacts 
from the WorkAdvance 
Demonstration 

Richard Hendra 
David H. Greenberg 
Gayle Hamilton 
Ari Oppenheim 
Alexandra Pennington 
Kelsey Schaberg 
Betsy L. Tessler 


mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 


June 2016 



PREVIEW SUMMARY 


Encouraging Evidence on a Sector-Focused 
Advancement Strategy 

Two-Year Impacts from the WorkAdvance Demonstration 


Richard Hendra 
David H. Greenberg 
Gayle Hamilton 
Ari Oppenheim 
Alexandra Pennington 
Kelsey Schaberg 
Betsy L. Tessler 


June 2016 



BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 


WorkAdvance is one of five evidence-based programs that were implemented as part of the Social 
Innovation Fund (S1F) grant to the Mayor’s Fund to Advance New York City and the Center for 
Economic Opportunity. The S1F unites public and private resources to evaluate and grow innova- 
tive community-based solutions with evidence of results. The S1F is a program of the Corporation 
for National and Community Service (CNCS), a federal agency that engages more than 5 million 
Americans in service through its AmeriCorps, Senior Coips, Social Innovation Fund, and 
Volunteer Generation Fund programs, and leads the President’s national call to service initia- 
tive, United We Serve. 

Matching funds for WorkAdvance have been provided by Bloomberg Philanthropies, Open 
Society Foundations, The Rockefeller Foundation, Altman Foundation, Common Bond Founda- 
tion, Ford Foundation, The Fund for Our Economic Future, George Kaiser Family Foundation, 
The Ira W. DeCamp Foundation, JPMorgan Chase Foundation, The Kresge Foundation, Robin 
Flood Foundation, Surdna Foundation, Tiger Foundation, Tulsa Area United Way, Tulsa 
Community Foundation, and The Flarry and Jeanette Weinberg Foundation, Inc. 

Dissemination of MDRC publications is supported by the following funders that help finance 
MDRC’s public policy outreach and expanding efforts to communicate the results and implica- 
tions of our work to policymakers, practitioners, and others: The Annie E. Casey Foundation, 
Charles and Lynn Schusterman Family Foundation, The Edna McConnell Clark Foundation, Ford 
Foundation, The George Gund Foundation, Daniel and Corinne Goldman, The Flarry and Jeanette 
Weinberg Foundation, Inc., The JBP Foundation, The Joyce Foundation, The Kresge Foundation, 
Laura and John Arnold Foundation, Sandler Foundation, and The Starr Foundation. 

In addition, earnings from the MDRC Endowment help sustain our dissemination efforts. Con- 
tributors to the MDRC Endowment include Alcoa Foundation, The Ambrose Monell Foundation, 
Anheuser-Busch Foundation, Bristol-Myers Squibb Foundation, Charles Stewart Mott Founda- 
tion, Ford Foundation, The George Gund Foundation, The Grable Foundation, The Lizabeth and 
Frank Newman Charitable Foundation, The New York Times Company Foundation, Jan Nichol- 
son, Paul H. O’Neill Charitable Foundation, John S. Reed, Sandler Foundation, and The Stupski 
Family Fund, as well as other individual contributors. 

This material is based upon work supported by CNCS under Grant No. 10S1HNY002. Opinions 
or points of view expressed in this document are those of the authors and do not necessarily 
reflect the official position of, or a position that is endorsed by, CNCS or the Social Innovation 
Fund. 

The findings and conclusions in this report do not necessarily represent the official positions or 
policies of the funders, The Mayor’s Fund to Advance New York City, or the New York City 
Center for Economic Opportunity. 


For information about MDRC and copies of our publications, see our website: www.mdrc.org. 
Copyright © 2016 by MDRC®. All rights reserved. 



Overview 


Thi s report summarizes the two-year findings of a rigorous random assignment evaluation of the 
WorkAdvance model, a sectoral training and advancement initiative. Launched in 2011, WorkAd- 
vance goes beyond the previous generation of employment programs by introducing demand-driven 
skills training and a focus on jobs that have career pathways. The model is heavily influenced by the 
positive findings from the Sectoral Employment Impact Study (SE1S) completed in 2010. A major 
component of the WorkAdvance model, in common with the programs studied in the SE1S, is 
formal training offering industry-recognized certifications, reflecting the hypothesis that skill s 
acquisition is necessary for advancement. The model also requires providers to be far more employer- 
facing than traditional training programs, taking into account multiple employers’ changing skill 
requirements, employee assessment practices, and personnel needs. This report presents the imple- 
mentation, cost, participation, and two-year economic impacts of WorkAdvance. The economic 
results are based on unemployment insurance earnings records and a second-year follow-up survey. 

The WorkAdvance program operations and evaluation are funded through the federal Social 
Innovation Fund (S1F), a public -private partnership administered by the Coiporation for National 
and Community Service. This S1F project is led by the Mayor’s Fund to Advance New York City 
and the NYC Center for Economic Opportunity in collaboration with MDRC. 

Key Findings 

• All providers translated the WorkAdvance model into a set of concrete services, but it took time 
— more than a year for some components and providers — and a substantial amount of tech- 
nical assistance and support. As a result, at some sites, later study enrollees were more likely 
than earlier ones to experience a fully implemented and “mature” WorkAdvance program. 

• Overall, WorkAdvance resulted in very large increases in participation in every category of 
services, as well as in training completion and credential acquisition, compared with what would 
have happened in the absence of the program. Expenditures for the operation of WorkAdvance 
fell between $5,200 and $6,700 per participant at the four providers delivering the program. 

• WorkAdvance providers increased earnings, with variation in results that closely matched the 
providers’ experience in running sector-based programs and the extent to which the services 
they offered were demand driven. The most experienced sectoral provider, Per Scholas, had 
large and consistent impacts on both primary and secondary outcomes. Madison Strategies 
Group and Towards Employment, providers new to sectoral training, had promising but less 
consistent results that grew stronger for later enrollees. One provider, St. Nicks Alliance, did not 
produce positive impacts. The results did not differ dramatically across subgroups, though en- 
couragingly, WorkAdvance was able to increase earnings among the long-term unemployed. 

The evaluation as a whole provides important information for workforce development providers 
interested in pursuing a sector strategy. The analysis considers the role played by providers’ sector- 
specific training and preparation and the role played by the nature of the sectors themselves. Future 
priorities that emerge from the results are (1) understanding how to help the more disadvantaged access 
the programs and (2) learning how to build service capacity, given how complex the model is to run. 




Preview Summary 


Even in good economic times, many adults — particularly individuals with no more than a high 
school education — struggle to obtain and maintain jobs that pay enough to support their fami- 
lies and permit upward mobility. At the same time, some employers report difficulty finding 
employees with the right skills to meet their needs. This report summarizes the two-year effec- 
tiveness results of WorkAdvance — a workforce development program that seeks to meet the 
needs of participants and employers equally. For unemployed and low- wage working adults, the 
program provides occupational skills training in targeted sectors that have good quality jobs and 
room for advancement within established career pathways. The program’s strategy is a sector- 
based one, in which program management and staff members also seek to fully understand — 
and fulfill — the skill requirements and other needs of employers in the targeted sectors. 

In brief, the two-year WorkAdvance effectiveness findings are encouraging: A rigor- 
ous randomized controlled trial indicates that sectoral programs can increase earnings among 
low-income individuals. One provider had large and consistent impacts on both primary and 
secondary outcomes. Two providers, both of which were new to the sectoral strategy and 
gained experience over time, had promising, though less consistent, results that grew stronger 
for later entrants into the program. One provider did not produce positive impacts. But the 
results also show that these types of programs are hard to run well, and, even when they are 
implemented well, impacts take time to emerge. The evaluation as a whole provides im- 
portant information for workforce development providers interested in pursuing a sector strat- 
egy. The analysis considers the role played in the results by providers preparing participants 
for jobs in specific sectors and the role played by the nature of the sectors themselves. 

WorkAdvance has been implemented in diverse settings by four providers specializing 
in specific sectors in which they have sought to develop relationships with employers and in- 
depth industry knowledge: Per Scholas (in New York City) targeted the information technology 
(IT) sector; St. Nicks Alliance (also in New York City) focused on environmental remediation; 
Madison Strategies Group (in Tulsa, Oklahoma) focused on transportation and, later, manufac- 
turing; and Towards Employment (in northeast Ohio) targeted health care and manufacturing. 
Table 1 presents information on the types of training provided at each site. The providers dif- 
fered in a variety of ways. Two of the providers (Per Scholas and Madison Strategies Group) 
were solely focused on WorkAdvance and other sectoral training initiatives. By contrast, St. 
Nicks Alliance is a large multiservice organization with a small workforce division. Towards 
Employment runs a comprehensive employment program in addition to the WorkAdvance pro- 
gram, providing job placement and other services. The providers also differ significantly with 
regard to their experience. Per Scholas has been operating an IT sector program since 1998, and 
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The WorkAdvance Study 
Table 1 


Key Details of WorkAdvance Service Providers 


Site 

(Location) 

Per Schoias 
(Bronx, NY) 

St. Nicks Alliance 
(Brooklyn, NY) 

Madison Strategies 
Group 
(Tulsa, OK) 

Towards Employment 
(Cleveland, OH) 

Targeted 

sector(s) 

Information technology 

Environmental 

remediation 

Transportation, 

manufacturing 

Health care, 
manufacturing 

Study 
sample size 

690 

479 

697 

698 

Structure 

Single program 

Division of large 
multiservice program 

Single program 

Multiple employment 
programs 

Approach 

Training- first 

Training- first 

Mixed until fall 2012 
(training-first and 
placement-first) and 
later predominantly 
training-first 

Mixed until fall 2012 
(training-first and 
placement-first) and 
later predominantly 
training-first 

Occupational skills training 

Length 

15 weeks 

5 to 12 weeks 

4 to 32 weeks 

2 to 17 weeks 

Location 

On-site 

On-site for pest control 
training; mix of on- and 
off-site at private 
schools for other 
training courses 

Off-site at private or 
technical schools or 
community colleges 

Off-site at private or 
technical schools or 
community colleges 

Offerings 

A Plus, Network Plus 

Environmental 
Remediation, 
Commercial Driver's 

License B with hazmat 
endorsement, Pest 
Control 

Aviation 
Manufacturing, 
Commercial Driver's 
License A and B, 
Computerized 
Numerical Control 
(CNC) Machining, 
Diesel Mechanic, 
Welding, Supervisory 
Leadership 

Computerized 
Numerical Control 
(CNC) Machining, 
Welding, Phlebotomy, 
Certified Health Care 
Access Associate, 
Patient Care Assistant, 
State -Tested Nurse 
Assistant, Medical 
Billing and Coding 


SOURCES: MDRC sample size counts from the WorkAdvance baseline information form; other information 
from documentation supplied by providers and interviews with provider staff members. 


St. Nicks Alliance has given training in environmental remediation since 2001. By contrast, 
Madison Strategies Group was operating in a new city and Towards Employment was operating 
in a new sector (manufacturing). 

WorkAdvance’s programming and evaluation have been conducted under the auspices 
of the Social Innovation Fund (SIF). Administered by the Corporation for National and Com- 
munity Service, the SIF is a public-private partnership designed to identify and expand effective 
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solutions to critical social challenges. WorkAdvance is part of the New York City Center for 
Economic Opportunity (CEO) SIF project, which is led by CEO and the Mayor’s Fund to Ad- 
vance New York City in collaboration with MDRC. MDRC is leading the WorkAdvance eval- 
uation; has provided technical assistance to the providers; and, jointly with CEO, has monitored 
providers’ operations. Funding for the WorkAdvance program and evaluation came from the 
SIF and a broad array of local funding partners. 


The WorkAdvance Program Model 

The WorkAdvance model goes beyond the previous generation of employment programs by 
introducing sector-based skills training and a focus on jobs that have career ladders or pathways, 
rather than just seeking “any job” or an initial credential for participants. The model is heavily 
influenced by the positive findings from the Sectoral Employment Impact Study (SEIS) com- 
pleted in 2010. 1 A major component of the WorkAdvance model, in common with the pro- 
grams studied in the SEIS, is formal training offering industry-recognized certifications, reflect- 
ing the hypothesis that skills acquisition is necessary for advancement. Additionally, the Work- 
Advance model requires providers to be far more employer-facing than traditional training pro- 
grams, taking into account multiple employers’ changing skill requirements, employee assess- 
ment practices, and human resource needs. 

The essential theory behind WorkAdvance is that strategic, demand-guided upgrades in 
human capital — that is, education and employment-related skills and experience — will even- 
tually lead to advancement in the labor market. This theory informs the key components of the 
WorkAdvance model: 

1. Intensive screening of program applicants before enrollment — a practice 
not always found in employment programs offered to low-income individu- 
als — intended to ensure that the program providers select participants who 
can take advantage of the skills training in the sector and be qualified for 
specific occupations within it. 

2. Sector-appropriate preemployment and career readiness services, consist- 
ing of an orientation to the sector, career readiness training (which stresses 
how to search for work in the sector and how to comport oneself on the job), 
individualized career coaching, and limited supportive services to sustain 
program engagement and assist participants to complete their training and 
find employment. 


’Sheila Maguire, Joshua Freely, Carol Clymer, Maureen Conway, and Deena Schwartz, Tuning In to 
Local Labor Markets: Findings from the Sectoral Employment Study (Philadelphia: Public/Private Ven- 
tures, 2010). 
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3. Sector-specific occupational skills training, intended to impart skills and 
lead to credentials that will substantially enhance workers’ employment op- 
portunities. Training is geared toward current job openings in specific sectors 
and occupations, and training offerings are to be adapted to changes in avail- 
able jobs. (In the first half of the program enrollment period, the goal was to 
place approximately half the enrollees at two providers into a placement-first 
track, in which they skipped the initial training phase, so the model’s training 
component was not offered to all program enrollees.) 

4. Sector-specific job development and placement services, intended to facili- 
tate entry into positions for which participants have been trained and for 
which there are thought to be genuine opportunities for continued skill de- 
velopment and career advancement. The providers’ job developers are to 
maintain strong relationships with employers who hire individuals with the 
kinds of skills the program imparts. 

5. Postemployment retention and advancement services, provided to assist 
participants beyond the placement stage. Providers are to maintain close con- 
tact with workers and employers to assess performance, offer coaching to 
address any complicating life situations that might arise for workers, help 
identify next-step job opportunities and skills training to enable participants 
to move up career ladders over time, and assist with rapid reemployment if 
workers lose their jobs. It should be noted that any effects of these services 
may not yet be seen in this report given that only slightly over two years of 
follow-up data are available. 


Evaluation Design and Study Enrollee Characteristics 

The WorkAdvance programs at the four sites were studied using a random assignment design, a 
methodology often referred to as the “gold standard” in evaluation research, which allows prac- 
titioners and policymakers to have a very high degree of confidence in the results. From June 
2011 to June 2013, individuals who met the WorkAdvance eligibility criteria, as well as the re- 
quirements for the specific sector programs, were assigned at random to the WorkAdvance 
group or to the control group. Members of the WorkAdvance group were eligible to receive 
WorkAdvance services, while those in the control group were not eligible for these services but 
could get other services and support available in the community. Both research groups were 
tracked over time. The random assignment process ensured that, when individuals entered the 
study, there were no systematic differences in sample members’ characteristics, measured or 
unmeasured, between the two research groups. Thus, any differences between the groups that 
emerge after random assignment can be attributed to the WorkAdvance program. 
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During the evaluation period, WorkAdvance targeted unemployed and low-wage work- 
ing adults with a monthly family income below 200 percent of the federal poverty level. Adults 
who were working at the time they entered the study were required to be earning less than $15 
per hour. Sector-specific requirements might include drug testing, literacy or numeracy testing, 
or screens for criminal history, and in many cases individuals had to demonstrate that they could 
engage in training full time and that they had a high degree of interest in a career in the sector. 
Across the sites, 2,564 individuals were enrolled in the study and randomly assigned to either 
the WorkAdvance group (1,293) or the control group (1,271). Site-level sample sizes range 
from 479 at the St. Nicks Alliance site up to 698 at Towards Employment. 2 

Across the sites, the average WorkAdvance sample member was 34 years old, and the 
majority of sample members were male (73 percent) and single (67 percent). (These averages 
mask important variation by site and sector. For example, the health care sector at Towards 
Employment mostly enrolled females.) Half the participants were black/African-American, 1 8 
percent were white, and 17 percent were Latino/Hispanic. Almost all sample members had at 
least a high school diploma or equivalent, and over half the sample had at least some college 
education, though most did not have a college degree of any type. Only one in five were work- 
ing at the time they entered the study, and more than one-third of sample members had been 
unemployed for at least seven months immediately preceding study entry. Overall, 15 percent of 
sample members had work experience in their targeted industry. At study entry, less than half 
the enrollees were covered by health insurance, more than one-third were receiving Supple- 
mental Nutrition Assistance Program (SNAP) benefits (food stamps), 16 percent were receiving 
unemployment insurance benefits (even though most were unemployed), and 6 percent were 
receiving Temporary Assistance for Needy Families (TANF). Despite fairly high levels of edu- 
cation and work experience, many sample members still faced substantial barriers to employ- 
ment: One-quarter of enrollees, for example, had a criminal conviction, and even higher rates 
were seen among transportation and manufacturing sector enrollees. 

This report extends the evaluation findings of two previous documents, 3 addressing a 
number of research questions: How did the four providers implement the program model, and 
what adjustments did they make over time? To what extent did WorkAdvance increase en- 
gagement in employment and training services, compared with what would have happened in 
the absence of the program (the control group)? What does it cost to operate the WorkAdvance 
program? To what extent did WorkAdvance, over a two-year follow-up period, improve indi- 


2 Attrition from the study was minimal: Only 20 individuals (less than 0.8 percent of the original sample) 
withdrew from the study at any point following their random assignment. 

3 Betsy Tessler, WorkAdvance: Testing a New’ Approach to Increase Employment Advancement for Low- 
Skilled Adults (New York: MDRC, 2013), and Betsy L. Tessler, Michael Bangser, Alexandra Pennington, 
Kelsey Schaberg, and Hannah Dalporto, Meeting the Needs of Workers and Employers: Implementation of a 
Sector-Focused Career Advancement Model for Low-Skilled Adults (New York: MDRC, 2014). 
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victuals’ employment, earnings, income (including public assistance), and life satisfaction? And 
what helps explain the patterns in WorkAdvance’s economic effects — for example, what is the 
influence of program maturity? 

Sources of data analyzed in this report to determine WorkAdvance’s effectiveness in- 
clude a follow-up survey, in which the average survey respondent was interviewed 22 months 
following his or her random assignment (the “Year 2 Survey,” for which about 80 percent of the 
entire study sample was interviewed), and unemployment insurance data, available for nine 
quarters (2.25 years) following random assigmnent (obtained for all sample enrollees). Thus, all 
study enrollees have approximately two years of common follow-up, in most cases from both 
the survey and the unemployment insurance administrative records. 


Key Findings 

Analyses in this report yielded the following key findings. 

Implementation and Participation Findings 

• Translating the WorkAdvance model into a set of concrete services took 
time — more than a year for some components and providers — and a 
substantial amount of technical assistance and related support; some 
providers required more time and assistance than others. As a result, at 
some sites, a fully implemented and “mature” WorkAdvance program 
was more likely to be experienced by later study enrollees than by early 
study enrollees. 

All four providers eventually implemented all the WorkAdvance model components, 
with postemployment services the last to be fully developed and implemented. The engagement 
of WorkAdvance group members in career readiness services, occupational skills training, and 
postemployment retention and advancement services was high for all four providers, in part re- 
flecting the pre-random assigmnent intake process that screened out individuals who were not 
motivated or able to complete the intake process, who did not have the literacy levels needed to 
take full advantage of the training, or who had other impediments to being hired in the sector. 

While all WorkAdvance providers delivered the model components, differences in im- 
plementation across the providers were apparent. Per Scholas’s history of providing information 
technology training and its long-standing connections with employers in that sector gave it a 
head start on the other providers. Because it did not outsource skills training but provided it on 
site, Per Scholas staff members also had more opportunities to interact with program partici- 
pants, deliver career readiness services, and reinforce workplace behaviors. Madison Strategies 
Group and Towards Employment initially operated a dual-track approach, in which some enrol- 
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lees were first placed in jobs, before being offered training opportunities. This resulted in early 
enrollees at those two sites being less likely than enrollees at the other two sites to ever start and 
complete occupational skills training. These two providers were also the newest to sectoral 
training. For these reasons, later program enrollees (the “late cohort”) probably experienced a 
stronger program at those two sites. Finally, while all providers had ties to employers in their 
targeted sectors, some providers had more experience being demand driven than others or had 
closer ties to employers. Per Scholas had the most experience delivering demand-driven ser- 
vices (which means that employer input is evident in all program components). 

• Across all sites, WorkAdvance resulted in very large increases in partic- 
ipation in every category of services, as well as in training completion 
and credential acquisition, compared with what would have happened 
in the absence of the program. 

On their own initiative the control group members obtained a range of employment ser- 
vices, including (in some cases) training in the targeted sector, from community colleges and 
other providers. It is not surprising that the control group members found their way to services, 
given that they had to be quite motivated to persist through the WorkAdvance screening pro- 
cess. WorkAdvance, however, resulted in many more individuals participating in training, as 
well as in career readiness, job search, and postemployment services, over a two-year follow-up 
period, and the evaluation is thus in a good position to measure the economic value that these 
activities and services can add. For example, WorkAdvance increased participation in vocation- 
al training in the targeted sector by approximately 40 percentage points or more at every site, 
relative to control group members’ levels (which ranged from 13 percent to 21 percent depend- 
ing on the site). WorkAdvance also increased the likelihood of completing such training by 31 
percentage points or more (as shown in Figure 1) and increased the likelihood of attaining a vo- 
cational training credential in the targeted sector by 25 percentage points or more. These results 
are impressive, particularly given that two of the providers — Madison Strategies Group and 
Towards Employment — placed approximately half their early enrollees in a placement-first 
track. MDRC calculations indicate that expenditures on the operation of WorkAdvance were 
within the range of $5,200 to $6,700 per program group study participant at the four providers 
delivering the program. 

Economic Impact Findings 

• The WorkAdvance programs at all providers increased participants’ 
employment in the targeted sector, but the size of the impact varied sub- 
stantially across the sites. 
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Percentage 


The WorkAdvance Study 
Figure 1 

Completion of Training in the Targeted Sector, by Site 





Per Scholas St. Nicks Alliance Madison Strategies Towards Employment 

Group 

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Figure 2 shows that at all four sites WorkAdvance increased employment in the target- 
ed sector, relative to the experiences of control group members. Of all the sites, Per Scholas 
produced by far the largest impact on employment in the targeted sector: 61 percent of Work- 
Advance group members, when surveyed, reported working in the information technology sec- 
tor at their current or most recent job, a striking 41 percentage points higher than was the case 
for control group members. St. Nicks Alliance had the lowest rate of WorkAdvance group 
members who reported current or recent work in the targeted sector — 32 percent — but its im- 
pact of 12 percentage points on this measure is still statistically significant. Madison Strategies 
Group and Towards Employment both increased current or recent targeted sector employment 
by a little over 16 percentage points, even with relatively high control group rates. 


8 



The WorkAdvance Study 
Figure 2 

Employment in the Targeted Sector at Current or Most Recent Job, by Site 



Per Scholas St. Nicks Alliance Madison Strategies Towards Employment 

Group 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


• Impacts on earnings — from any type of job in any sector — varied 
across the sites, in a pattern that closely matched the providers’ experi- 
ence in running sector-based programs and the extent to which the ser- 
vices they offered were demand driven. 

Figure 3 shows impacts on earnings in Year 2 of the follow-up period. Early in the fol- 
low-up period, WorkAdvance group members’ earnings levels were lower than those of the 
control group owing to the well-known labor market opportunity cost of training participation. 
This situation, however, changed for three of the four providers in Year 2. At Per Scholas 
(which has long operated an information technology sector training program), WorkAdvance 
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The WorkAdvance Study 
Figure 3 

Impacts on Earnings in Year 2, by Site 
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SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York 
State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio Department of Job and 
Family Services for Towards Employment sample members; and Oklahoma Employment Security Commission for 
Madison Strategies Group sample members. 

NOTES: WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


increased earnings by over $3,700 (or 26 percent) above the control group level in Year 2. At 
Madison Strategies Group (which was a spin-off from an experienced provider that sought to 
operate programs in a new area of the country) and Towards Employment (an organization 
relatively new to working in one of the sectors in which it operated for WorkAdvance), statis- 
tically significant impacts on earnings began to emerge in Year 2, particularly toward the end 
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of that year. At St. Nicks Alliance, statistically significant impacts on earnings had not yet 
emerged by Year 2. 4 

• At the two sites where many early enrollees were channeled into a 
placement-first track, impacts were stronger for the late enrollee cohort 
than for the early enrollee cohort, as the providers matured in their abil- 
ity to deliver services and switched to a mostly training-first model. 

Figure 4 shows the impacts of WorkAdvance by random assigmnent cohort — an anal- 
ysis that in the evaluation’s planning stage was seen as critically important. Two of the pro- 
grams (Madison Strategies Group and Towards Employment) both initially implemented a 
mixed model, in which the goal was to place half the program participants into jobs in the tar- 
geted sectors before training. As shown in Figure 4, at both providers impacts were dramatically 
stronger for individuals who enrolled in the second half of the intake period than for individuals 
who enrolled in the first half. It is difficult to determine reliably the extent to which the better 
results for the later cohort were due to a greater emphasis on training versus an overall matura- 
tion of the program at these providers, both of which were new to sectoral programming. 

The cohort analysis is the least relevant to Per Scholas, as it was the most mature pro- 
vider coming into the study. However, the results indicate that program effects were somewhat 
weaker for the later cohort at that site than for the early cohort; the reasons are unclear, but these 
results could relate to the strengthening local economy during the study period (and perhaps to 
other factors as well). At St. Nicks Alliance, no improvement in impacts was evident for the 
later cohort relative to the early cohort. 

• The program at Per Scholas produced impacts on the widest array of 
secondary employment outcomes. However, most providers produced 
impacts on some indicators of career advancement or advancement 
potential. 

The WorkAdvance program at Per Scholas increased hourly wages and weekly earn- 
ings, and the Madison Strategies Group program increased the likelihood of participants 


4 As discussed in the upcoming full report, two measures were prespecified as “confirmatory” measures: 
earnings in the last quarter available for the full sample (which originally was planned as Quarter 7, but was 
changed to Quarter 10 after data collection was extended) and employment (in any sector) at the time of the 
survey. Only the Per Scholas program produced statistically significant impacts if just these two measures are 
considered. Madison Strategies Group and Towards Employment produced impacts that are not quite statisti- 
cally significant on these measures, though the impacts on earnings in Quarter 10 are statistically significant 
when those two sites are pooled. St. Nicks Alliance did not produce impacts on either measure. 
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The WorkAdvance Study 
Figure 4 

Impacts on Earnings in Year 2, by Site and Random Assignment Cohort 
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SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York State Department of Labor for Per Scholas and 
St. Nicks Alliance sample members; Ohio Department of Job and Family Services for Towards Employment sample members; and Oklahoma Employment 
Security Commission for Madison Strategies Group sample members. 


NOTES: WA group = WorkAdvance (program) group. 

The early cohort includes sample members randomly assigned through Quarter 3, 2012. The late cohort includes all sample members randomly assigned in 
and after Quarter 4, 2012. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 



working at jobs with higher wages. WorkAdvance had no effect on average wages or earnings 
in the current or most recent job (as of the survey administration) at the other sites and did not 
affect the likelihood of working full time at any of the sites. 

At all sites except for St. Nicks Alliance, WorkAdvance group members were more 
likely than their control group counterparts to report that the job they held currently or most re- 
cently as of the survey administration offered many opportunities for career advancement. 
WorkAdvance also improved other aspects of job quality and advancement. Notably, WorkAd- 
vance increased the proportion of individuals who received several types of employment-related 
benefits at Madison Strategies Group; improved job satisfaction at both that site and Per Scho- 
las; and increased the proportion of people who reported working in a regular permanent job, as 
opposed to a temporary one, at Towards Employment. 

The impact analysis also examined a range of measures related to income, material 
hardship, and overall well-being. Only WorkAdvance at Per Scholas produced impacts in all 
these domains. The Per Scholas program increased income, reduced material hardship, reduced 
public assistance usage, and increased overall life satisfaction. It is unusual to see such a con- 
sistent pattern of impacts across so many domains. At the other sites, only a few impacts on 
such measures are statistically significant. 

• The extent to which WorkAdvance increased employment in targeted 
sector jobs was the critical factor in explaining the pattern of impacts 
across the sites. At all the sites, jobs in the targeted sector were generally 
of higher quality than jobs outside the targeted sector. 

For a sectoral program to produce employment and earnings impacts, the program 
needs to increase employment in the targeted sector, and the targeted sector jobs have to have 
better characteristics than jobs outside the targeted sector (that is, there needs to be a “sector 
premium”). In terms of increasing targeted sector employment, Per Scholas was by far the most 
successful. As shown in Figure 2, the 41 percentage point impact on targeted sector employ- 
ment at that site was more than twice as large as the impact at any of the other sites and almost 
four times as large as the impact at St. Nicks Alliance. This differential in targeted sector em- 
ployment appears to explain why Per Scholas produced the strongest impacts and why St. Nicks 
Alliance produced the weakest impacts. 

At most sites, an examination of the jobs held by WorkAdvance group members who 
worked following random assignment suggests that targeted sector jobs were better than jobs 
outside the targeted sector. At Per Scholas, St. Nicks Alliance, and Madison Strategies Group, 
targeted sector jobs paid more than other jobs that WorkAdvance group members found. In ad- 
dition, at most sites, the typical number of hours required by the jobs was higher, or other job 
characteristics, such as offered benefits, tended to be better, for jobs WorkAdvance group 
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members held in the targeted sector than for jobs they held outside the targeted sector. Thus, 
there was no notable variation among the sites in the “premium” added for targeted sector jobs 
versus non-targeted sector jobs. Instead, the overriding factor for why the impacts varied from 
site to site appears to be differences in the extent to which the sites increased placements in the 
targeted sector. 

• The results also highlight that sector programs can be hard to run well. 

The WorkAdvance model was intended to go well beyond traditional workforce devel- 
opment models by incorporating a sectoral and advancement emphasis. It will take time for 
many providers to develop these capacities. In particular, the St. Nicks Alliance WorkAdvance 
program confronted numerous difficulties in adapting its more traditional vocational training 
program to the Work Advance model, which may explain why impacts have not emerged, at 
least through this report’s follow-up period. St. Nicks Alliance is a highly experienced commu- 
nity-based multiservice provider with a relatively small workforce division. The WorkAdvance 
program at St. Nicks Alliance experienced a collapse in the demand for environmental remedia- 
tion work early in the program period and faced challenges in responding to these changes. A 
more effective response would have required a more proactive approach with employers than 
St. Nicks had previously used. 

• Pooling the results from the four providers, the economic impacts of 
WorkAdvance are positive and statistically significant, but this masks 
considerable variation at the site level. 

During the evaluation’s analysis planning phase, a decision was made to focus the im- 
pact analysis at the site level, 5 given the providers’ widely varying experience with WorkAd- 
vance-like services and sectoral strategies. Still, it is instructive for policymakers to understand 
how a strategy like WorkAdvance might perform overall, across a range of providers and loca- 
tions. Figure 5 shows, for all the sites pooled together (left set of columns), WorkAdvance’s 
impact on earnings in the post-training period (Year 2). As a whole, WorkAdvance increased 
earnings by about $1,945, or about 14 percent, above the control group level in Year 2. The re- 
sults show that with a configuration of sites similar to those evaluated in WorkAdvance, poli- 
cymakers can expect positive impacts on earnings overall. It is important, however, to recognize 
that this overall effect will mask considerable variation at the site level. For example, without 
Per Scholas, the pooled impacts are less than half as large, though still statistically significant. 


initially, the analysis plan called for an examination of impacts by sector. This changed after a site was 
dropped from the study, because the sector results would have lacked sufficient statistical power. The decision 
to examine impacts by site was made well before impact results were available. 
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The WorkAdvance Study 
Figure 5 

Impacts on Earnings in Year 2 for All Sites Pooled, 
by Prior Attachment to the Labor Market 
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SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York 
State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio Department of Job and 
Family Services for Towards Employment sample members; and Oklahoma Employment Security Commission for 
Madison Strategies Group sample members. 

NOTES: WA group = WorkAdvance (program) group. 

The fully attached subgroup consists of sample members who were working at random assignment or who were 
unemployed for less than one month before random assignment. The semiattached subgroup consists of sample 
members who were unemployed for one to six months before random assignment. The long-term unemployed 
subgroup consists of sample members who have never been employed or who were unemployed for seven or more 
months before random assignment. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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• WorkAdvance produced positive impacts for the long-term unem- 
ployed. 

WorkAdvance operated during the long wake of the Great Recession of 2007-2009. 
During this period, the number of people who qualified as long-term unemployed increased 
markedly, and there was significant concern about the likelihood of reengaging this group in the 
labor market. Given this concern, the analysis examined WorkAdvance’s effects for subgroups 
defined by individuals’ prior level of attachment to the labor market (Figure 5). This analysis 
found that WorkAdvance produced statistically significant impacts on employment and earn- 
ings for both the long-tenn unemployed and those who were semiattached to the labor market. 6 


Conclusions 

• The WorkAdvance results show that sectoral programs can increase 
earnings among low-income individuals. But even when a program is 
well implemented, the benefits take time to emerge, for providers that 
have lengthy experience in sectoral training as well as for providers that 
are new to the strategy. 

The Per Scholas program has now been shown to produce large impacts in two separate 
studies: in this study as well as in the Sector Employment Impact Study (SEIS). 7 The WorkAd- 
vance results thus provide important validation that a mature and highly functioning sectoral 
training provider can produce large impacts on a consistent basis. More generally, the results 
provide confinnation of the SEIS results, which showed that three separate providers (including 
Per Scholas) produced large effects on low-income individuals’ earnings. The WorkAdvance 
impacts at Per Scholas are evident using both survey data and unemployment insurance records 
and across a large number of primary and secondary outcomes. Moreover, the size and con- 
sistency of the impacts of Per Scholas’s sector training program are unusual in the history of 
employment and training studies. 

The WorkAdvance results also highlight that in sectoral training programs, impacts take 
time to emerge, for two reasons. First, for participants, it takes time for the training and ad- 


6 For this analysis, long-term unemployment was defined as never having been employed or having been 
unemployed for seven months or more immediately preceding study entry. Note that the more positive results 
among the long-term unemployed are related to the fact that sites that served more long-term unemployed 
workers had larger impacts in general. Therefore, while the results show that WorkAdvance can work for the 
long-term unemployed, the results do not necessarily imply that the program will always work better for that 
group, nor do they demonstrate that the program will be less effective among the recently or currently em- 
ployed. In general, a variety of analyses suggest that the characteristics of sample members in each site did not 
explain the variation in impacts across the sites. 

Maguire et al. (2010). 
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vancement services to be completed and for them to translate into job placements and work ad- 
vancement. Even for individuals enrolled at the experienced Per Scholas, impacts did not 
emerge until the second year of follow-up. Second, for providers new to a sector, it takes time 
for program staff relationships with employers and staff specialization in sector practices to take 
root and to result in strong and tailored training programs. Both these factors were clearly in 
play at Madison Strategies Group and Towards Employment, two providers that implemented 
WorkAdvance well, but which were new to their sector or location. At these two sites, impacts 
on participants’ earnings emerged only in the second year of follow-up. In addition, the impacts 
were much stronger for the late-enrolling cohorts, who experienced stronger programs. 

It is likely that the wide range of experience and backgrounds among the WorkAdvance 
providers is typical of the landscape of workforce development providers across the nation, and 
thus the WorkAdvance findings have important implications for Workforce Innovation and Op- 
portunity Act (WIOA) implementation and workforce practice in general. (Many of these im- 
plications are discussed in the concluding chapter of the upcoming full report.) In this sense, the 
WorkAdvance evaluation provides the field with a reliable indication of how the sectoral strate- 
gy will work among a representative range of providers. When all sites are pooled together, the 
results of WorkAdvance are positive and statistically significant. This implies that, on average, 
the programs can be effective. But the report results also show that the size and particularly the 
timing of impacts may vary critically based on where providers start from in terms of their ex- 
perience in the sector. The sectoral strategy requires highly capable providers, and the report 
results made it clear that not all providers will produce positive impacts via the sectoral training 
approach. The results also highlight that for the sectoral and postemployment advancement 
strategy featured in WorkAdvance to translate into earnings gains, providers need to focus on 
increasing targeted sector employment in jobs that have higher wages and better characteristics 
than the jobs participants can find on their own in other sectors. This, in turn, will happen only if 
program services are truly demand driven, which requires that program leaders and staff mem- 
bers at all levels understand many aspects of their targeted sector — its language, needs, met- 
rics, and skill requirements. 

* * * 

While the upcoming full report, summarized here, presents the final WorkAdvance im- 
pact estimates under the project’s Social Innovation Fund grant, additional reports targeted to 
specific audiences are planned. Future publications will consider the implications of the Work- 
Advance experience for practitioners and for the career pathways field in general. Furthermore, 
longer follow-up at the three- and five-year points is under consideration. If this is undertaken, 
the results will provide important insights into whether impacts grow over time and about the 
effects of postemployment advancement services. If additional follow-up data are obtained, a 
full benefit-cost analysis will also be conducted. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



